Transforming logical data objected for storage includes identifying multiple write request to the same logical object and grouping transformed data chunks

ABSTRACT

There is provided a system capable of transformation of logical data objects for storage and method of operating thereof. The method comprises: a) identifying among a plurality of requests addressed to the storage device two or more “write” requests addressed to the same logical data object; b) deriving data chunks corresponding to identified “write” requests and transforming the derived data chunks; c) grouping the transformed data chunks in accordance with the order the requests have been received and in accordance with a predefined criteria; d) generating a grouped “write” request to the storage device; and e) providing mapping in a manner facilitating one-to-one relationship between the data in the obtained data chunks and the data to be read from the transformed logical object. The method further comprises obtaining an acknowledging response from the storage device; multiplying the obtained acknowledging response, and sending respective acknowledgements to each source which initiated respective “write” request.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application relates to and claims priority from U.S. ProvisionalPatent Application No. 61/163,267 filed on Mar. 25, 2009 incorporatedherein by reference in its entirety.

FIELD OF THE INVENTION

This invention relates to computing systems, and, in particular, tomethods and systems capable of transforming logical data objects to bestored in computing systems and networks thereof.

BACKGROUND OF THE INVENTION

With increasing demand for faster, more powerful and more efficient waysto store information, optimization of storage technologies has become akey challenge.

Logical data objects (data files, image files, data blocks, etc.) may betransformed for transmission and/or storage. The transformation maycomprise compression, encryption, encoding, conversion, etc. and/orcombinations thereof. For example, data compression techniques are usedto reduce the amount of data to be stored or transmitted in order toreduce storage capacity and transmission time respectively. Compressionmay be achieved by using different compression algorithms known in theart. Data stored in plaintext is open to potential malicious use (e.g.unauthorized access, misuse, theft, etc.), and known in the artsolutions for perimeter and/or access control (e.g. firewalls, VirtualPrivate Networks, LUN masking control and zoning in SAN storagenetworks, NAS security control features, etc.) still leave securityvulnerabilities. Encrypting data to be stored may considerably reducesecurity threats; such encryption may be provided by using differentalgorithms known in the art.

The problems of compressing, encrypting or other transforming data forstorage and access to the stored transformed data have been recognizedin the Prior Art and various systems have been developed to provide asolution. For example, various solutions are disclosed in InternationalApplications WO2007138600, WO2007138601 and WO2007138602 published Dec.12, 2007 and assigned to the assignee of the present application. Theentire contents of said International Applications are herebyincorporated by reference in their entirety.

SUMMARY OF THE INVENTION

In accordance with certain aspects of there present invention, there isprovided a method of operating a transformation system. Thetransformation system is operatively coupled to a plurality of clientsand to at least one storage device and operable to write a plurality ofdata chunks to a plurality of transformed logical data objects (LOs)stored in the storage device. Each given transformed LO corresponds to arespective original logical data object (LO) and is configured tosequentially accommodate a plurality of variable size data chunkscorresponding to the original LO, transformed by the transformationsystem and accommodated in accordance with an order in which said chunksare received. The method comprises: (a) identifying among a plurality ofrequests addressed to the storage device two or more “write” requestsaddressed to the same logical data object; (b) deriving data chunkscorresponding to said identified “write” requests and processing thederived data chunks, thereby giving rise to transformed data chunks; (c)grouping the transformed data chunks in accordance with the order therequests have been received, said grouping provided in accordance with apredefined criteria, thus giving rise to a group of transformed datachunks; (d) generating a grouped “write” request to the storage devicefor accommodating said group of transformed data chunks in therespective transformed LO whilst keeping the order the data chunks havebeen received; and (e) enabling mapping between the data in the originalLO and the data accommodated in the transformed LO in a mannerfacilitating one-to-one relationship between the data in the deriveddata chunks and the data to be read from the transformed logical object.

In accordance with further aspects of the present invention, the methodfurther comprises: (a) obtaining an acknowledging response from thestorage device; (b) multiplying the obtained acknowledging response, andc) sending respective acknowledgements to each source which initiated atleast one identified “write” request corresponding to the group oftransformed data chunks.

In accordance with further aspects of the present invention, at leastpart of the identified “write” requests may be obtained by interceptingby the transformation system a plurality of requests addresses by one ormore clients among said plurality of clients to said at least onestorage device and/or may be generated by the transformation systemduring a maintenance and/or optimization process.

In accordance with other aspects of the present invention, there isprovided a method of writing data to a transformed logical data objectstored in a storage device operable with at least one storage protocol.The transformed logical data object comprising a header, an indexsection and one or more accommodation sections with predefined size,said sections sequentially accommodating transformed data chunksresulting from transformation of a respective logical data object. Themethod comprises: (a) identifying among a plurality of requestsaddressed to the storage device two or more “write” requests addressedto the same logical data object; (b) deriving data chunks correspondingto said identified “write” requests and processing the derived datachunks, thereby giving rise to transformed data chunks; (c) grouping thetransformed data chunks in accordance with the order the requests havebeen received, said grouping provided in accordance with a predefinedcriteria, thus giving rise to a group of transformed data chunks; (d)generating a grouped “write” request to the storage device forsequentially accommodating said group of transformed data chunks inaccordance with the order these and previous chunks were received; and(e) updating the index section in a manner facilitating one-to-onerelationship between the data in derived data chunks and the data to beread from the transformed logical object. The operations are provided bya transformation system operatively coupled to a plurality of clientsand to one or more storage device. At least part of the identified“write” requests may be obtained by intercepting by the transformationsystem a plurality of requests addresses by one or more clients amongsaid plurality of clients to one or more storage devices and/or may begenerated by the transformation system during a maintenance and/oroptimization process.

In accordance with further aspects of the present invention, the methodfurther comprises: (a) obtaining an acknowledging response from thestorage device; (b) multiplying the obtained acknowledging response, and(c) sending respective acknowledgements to each source which initiatedat least one identified “write” request corresponding to the group oftransformed data chunks.

In accordance with other aspects of the present invention, there isprovided a transformation system capable of transforming a logical dataobject for storage in a storage device, the system operatively coupledto a plurality of clients and to at least one storage device configuredto accommodate data transformed by the transformation system. The systemcomprises: (a) means for creating in said at least one storage device aplurality of transformed logical data objects (LOs), each giventransformed LO corresponding to a respective original logical dataobject (LO) and configured to sequentially accommodate a plurality ofvariable size data chunks corresponding to the original LO, transformedby the transformation system and accommodated in accordance with anorder in which said chunks are received, said accommodation furthercomprising a mapping between the data in the original LO and the dataaccommodated in the transformed LO; (b) means for identifying among aplurality of requests addressed to the storage device two or more“write” requests addressed to the same logical data object; (c) meansfor deriving data chunks corresponding to said identified “write”requests and processing the derived data chunks, thereby giving rise totransformed data chunks; (d) means for grouping the transformed datachunks in accordance with the order the requests have been received,said grouping provided in accordance with a predefined criteria, thusgiving rise to a group of transformed data chunks; (e) means forgenerating a grouped “write” request to the storage device forsequential accommodation of transformed data chunks in respectivetransformed logical object; and (f) means for enabling updating themapping in a manner facilitating one-to-one relationship between thedata in the obtained data chunks and the data to be read from thetransformed logical object.

The system further comprises (a) means for obtaining an acknowledgingresponse from the storage device; and (b) means for multiplying theobtained acknowledging response, and sending respective acknowledgementsto each source which initiated at least one identified “write” requestcorresponding to the group of transformed data chunks.

The system further comprises means for intercepting at least part of theplurality of requests addresses by one or more clients among saidplurality of clients to said at least one storage device.

In accordance with further aspects of the present invention, the systemmay comprise a maintenance unit operable to generate during maintenanceand/or optimization process one or more “write” requests addressed tothe storage device.

Among advantages of certain embodiments of the present invention isreducing the amount of input/output operations executed by the storagesystem and required for write operations in accordance with obtained“write” requests.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it may be carriedout in practice, embodiments will now be described, by way ofnon-limiting example only, with reference to the accompanying drawings,in which:

FIG. 1 illustrates a schematic block diagram of typical storage networkarchitecture as is known in the art;

FIG. 2 illustrates a schematic functional block diagram of thetransformation system in accordance with certain embodiments of thepresent invention;

FIG. 3 illustrates a generalized flowchart of a write operation inaccordance with certain embodiments of the present invention; and

FIG. 4 illustrates a generalized flowchart of handling a write requestin a write request queue in accordance with certain embodiments of thepresent invention.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the invention.However, it will be understood by those skilled in the art that thepresent invention may be practiced without these specific details. Inother instances, well-known methods, procedures, components and circuitshave not been described in detail so as not to obscure the presentinvention. In the drawings and descriptions, identical referencenumerals indicate those components that are common to differentembodiments or configurations.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “computing”,“calculating”, “determining”, “generating”, “writing” or the like, referto the action and/or processes of a computer that manipulate and/ortransform data into other data, said data represented as physical, suchas electronic, quantities and/or said data representing the physicalobjects. The term “computer” should be expansively construed to coverany kind of electronic system with data processing capabilities,including, by way of non-limiting example, personal computers, servers,computing system, communication devices, storage devices, processors(e.g. digital signal processor (DSP), microcontrollers, fieldprogrammable gate array (FPGA), application specific integrated circuit(ASIC), etc.) and other electronic computing devices.

The operations in accordance with the teachings herein may be performedby a computer specially constructed for the desired purposes or by ageneral purpose computer specially configured for the desired purpose bya computer program stored in a computer readable storage medium.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the inventions as described herein.

The references cited in the background teach many principles oftransforming data for storage and operating the transformed data thatare applicable to the present invention. Therefore the full contents ofthese publications are incorporated by reference herein for appropriateteachings of additional or alternative details, features and/ortechnical background.

The term “logical data object (LO)” used in this patent specificationshould be expansively construed to include any types and granularitiesof data units used in a computing system and handled as one unit (e.g.data files, archive files, image files, database files, memory datablocks, stream data blocks, predefined segments of files, etc.).

Bearing this in mind, attention is drawn to FIG. 1 illustrating aschematic diagram of storage network architecture applicable to certainembodiments of the present invention. The logical data objects (LO) fromclients (workstations, application servers, etc.) 11 are transferred viaa network 12 to storage device(s) 13 (e.g. specialized NAS file servers,general purpose file servers, SAN storage, stream storage device, etc.)and/or 13 a. The network comprises one or more communication devices 14(e.g. switch, router, bridge, etc.) facilitating the data transfer. Thestorage in the illustrated network may be wholly or partly implementedusing block mode access and/or file mode access storage protocols. Infile mode access the logical data objects (LOs) are constituted byfiles, and the network is IP network (e.g. local area network (LAN),wide area network (WAN), combination thereof, etc.). In block modeaccess embodiments, the logical data objects are constituted by datablocks and the network is Storage Area Network (SAN) implementing, forexample, Fiber Channel or iSCSI protocols. In certain embodimentsclients may be directly connected to the storage device (13 a) via LANpart of the network 12, e.g. via block mode access storage protocols asSCSI, Fiber Channel, etc. Such Direct Access Storage may include, forexample, the internally attached local disk drives or externallyattached RAID (redundant array of independent disks) or JBOD (just abunch of disks), etc.

At least part of the logical data objects (LOs) may be stored in atransformed form (e.g. a compressed form and/or an encrypted form). Thetransformation or part thereof may be provided by a transformationsystem 16 operatively coupled to the clients and to the storagedevice(s) and configured to intercept LO call operations (LOaccess-related requests) as well as some control transactions and to actas a proxy on certain transactions. By way of non-limiting example, thetransformation system may be configured to transfer such control-relatedrequests as format disk, de-fragment disk, take a snapshot, etc. betweenthe client and the storage device in a transparent manner, whileintervening in data access-related transactions (e.g. read, write, etc.)and some control related transactions (e.g. capacity status, set end ofLO, etc.)

The transformation system 16 is capable of deriving and processing datacorresponding to the intercepted LO access-related request,facilitating, when appropriate, communication with and/or within thefile system for storing the processed data at the storage medium as atleast one transformed LO.

During creating a transformed LO and/or “write” request to thetransformed LO, the transformation system 16 derives data correspondingto said LO, transforms the data and facilitates writing at the storagedevice 13. A “read” operation proceeds in reverse direction; therequired LO or parts thereof is retrieved by the transformation system,de-transformed (partly or entirely, in accordance with required datarange and implemented transformation technique) and sent to theappropriate client.

Typically the secure keys used for encryption (if required) are heldseparately from the device providing encryption and/or storage, forexample, they may be held at a key holding platform 15 operativelycoupled with the transformation system 16. Likewise, coding tables andsimilar external data involved in the transformation process may be heldseparate from the processing and/or storage devices. When appropriate,the transformation system is capable to communicate with the externalplatform (e.g. keys holding platform 15) for obtaining external datainvolved in the transformation process (e.g. secure keys and/or securevalues or other metadata thereof involved in the transformation).

Non-limiting examples of the transformation system 16 are disclosed inInternational Applications WO2007138600, WO2007138601 and WO2007138602published Dec. 12, 2007 and assigned to the assignee of the presentapplication.

In certain embodiments of the invention, the transformation system 16may comprise maintenance unit 17 responsible for various maintenancetasks such as, for example, storage space optimization.

Note that the invention is not bound by the specific architecturedescribed with reference to FIG. 1. For purpose of illustration only,the following description is made with respect to processing by thetransformation system 16 the logical data objects to be stored at thestorage device, writing the processed data to the storage device 13,reading the data to be de-transformed (e.g. decrypted, decompressed,etc.) from the storage device, and de-transforming them by thetransformation system 16. Those versed in the art will readilyappreciate that the invention is, likewise, applicable to any purposedevice adapted to implement the teachings of the present invention andoperatively located on a physical and/or logical route between a clientsending a data access request to the logical data object and a storagelocation of the appropriate data, including embodiments wherein thetransformation system is integrated, fully or partly, with thecommunication device, workstation, application server and/or storagedevice.

FIG. 2 illustrates a schematic functional block diagram of thetransformation system 16 in accordance with certain embodiments of thepresent invention. The transformation system comprises a ClientInput/Output (I/O) block 21 coupled to a session manager 22. The clientI/O block is operable to forward data access-related requests (e.g.read, write, set end of file/truncate, etc.) received from the clientsor from the maintenance unit 17 to the session manager.

A session starts by access request to a logical data object (e.g. LUNcapacity request as, for example, SCSI LUN capacity request command;open file request, etc.) and ends by disconnect request (e.g. “LUNdisconnect”, “close file”, etc.) received from the same IP address(user) with regard to the same logical object. The session may handleone or more access requests addressed to the same logical data object(e.g. from multiple applications running on the same client, multiplerequests from certain single applications, etc.).

The session manager 22 is operable to derive and/or generatesession-related data (e.g. source session address, session counters,session status, relevant instances, etc.) and to hold the respectivedata in a memory unit 29. The session manager is further operable tohandle blocking all the relevant resources when the logical data objectis open and releasing said resources on disconnect. The session manageris further configured to recognize and manage a plurality of sessionsaddressed to the same logical data object handling one or more accessrequests.

The requests from one or more clients and/or requests from themaintenance unit 17 are transferred from the session manager to adispatcher 23 operatively coupled to the session manager. The dispatcher23 is operatively coupled to a logical data object manager 24, a buffermanager 25 and a transformation unit 26. The dispatcher 23 communicateswith the logical data object manager 24 for data related transactions(e.g. Read, Write, set end of file, etc.) and with the transformationunit 26 for providing transformation operations in accordance withcertain embodiments of the present invention.

The transformation unit is further operatively coupled to the memoryunit 29 and operable to compress, encrypt and/or otherwise transformdata, and to enable sending the transformed data to a physical diskthrough a storage I/O interface 28. The transformation unit is furtheroperable to enable reading data from the physical disk through thestorage I/O interface, to de-transform (e.g. decrypting and/ordecompressing) the respective buffer(s) and, optionally, to segmentand/or combine original and/or partly transformed data chunks during theprocessing. The transformation unit may comprise one or moretransformation blocks responsible for certain transforming operations(e.g. compression-decompression block 26-1 operatively coupled to theencryption/decryption block 26-2), and may be configured to facilitatedata transfer and necessary synchronization between the blocks.

The transformation unit 26 may be further configured to communicate withone or more external platforms storing external information related todata involved in the transformation process (e.g. the secure keys forreceiving the keys and/or metadata thereof); to receive the information,extract or generate the necessary data (e.g. key ID) and to managethereof. The received information may be temporary accommodated in atrusted memory within the transformation system, wherein thetransformation unit may provide a management of the information (e.g. tomanage accommodation of certain keys in the memory for certain timeperiod in accordance with a certain policy). In certain embodiments ofthe invention the encryption/decryption block 26-2 may further generateone or more encryption initialization vectors to be used for encryption(e.g. together with secure keys).

The logical object manager 24 is further operatively coupled to thememory unit 29 and is operable to handle ordering and memory sharing bydifferent logical data objects and parts thereof.

The logical object manager 24 is further operatively coupled to thememory unit 29 and comprises a pending writes unit 24-1 and a requestmapping unit 24-2, both units operatively coupled to the transformationunit 26. As will be further detailed with reference to FIGS. 3-4, thelogical object manager is further involved in grouping multiple accessrequests addressed to the same logical data object.

The pending writes unit 24-1 is operable to hold information of pendingwrite requests. The request mapping unit 24-2 is responsible forgrouping respective “write” requests and generating a grouped request tobe sent to the storage I/O interface 28. The request mapping unit 24-2is further responsible for handling a write response as will be furtherdetailed with reference to FIG. 3.

The buffer manager 25 is operable to manage memory buffer resources andis responsible for allocating and releasing memory buffer for operationsof other blocks. The transformation system further comprises anintegrity manager 27 operatively coupled to the session manager, thebuffer manager and the logical object manager. The integrity manager isresponsible for synchronization and general control of the processes inthe transformation system as, for example keeping the integrity of thelogical data objects, etc. It is further responsible for flashing thememory buffer to the physical disk(s) through the storage physical I/Ointerface 28, and reading when needed from the disk(s).

Those skilled in the art will readily appreciate that the invention isnot bound by the configuration of FIG. 2; equivalent and/or modifiedfunctionality may be consolidated or divided in another manner and maybe implemented in software, firmware, hardware, or any combinationthereof.

FIG. 3 illustrates a generalized flowchart of write operation providedto the logical data object (LO) in accordance with certain embodimentsof the present invention.

The transformation system receives 30 a plurality of “write” requests,each addressed to a respective certain LO and identifying the offset inthe respective LO and the range L of data to write. The “write” requestsmay be addressed by client(s) 11 to the storage device 13 andintercepted by the transformation system 16. Alternatively oradditionally, the “write” requests may be initiated (e.g. in accordancewith a predefined schedule and/or event) by the maintenance unit 17 oran external maintenance unit and addressed to the transformation system.

In accordance with certain embodiments of the invention, eachtransformed logical data object stored in the storage device comprises aheader; the header accommodates unique identification (ID) of thetransformed LO. The ID may be generated by the transformation systemwhen creating the transformed LO. Optionally, the ID may be furthermodified as detailed, for example in U.S. Provisional Application No.61/294,662 filed on Jan. 13, 2010, assigned to the assignee of thepresent application and incorporated hereby by reference in itsentirety. The transformation system generates ID in a mannerfacilitating one-to-one relationship between a logical data object andrespective transformed logical data object generated by thetransformation system and stored in the storage device. Accordingly, IDcomprised in the header of generated transformed LO characterizes alsothe respective logical data object. The transformation system is furtherconfigured, responsive to intercepting a request addressed to certainLO, to enable identifying respective stored transformed LO in a mannertransparent to a client.

ID of the transformed LO may be generated in accordance with one or moreparameters characterizing the logical data object, and selection of suchparameters may depend on a storage protocol. By way of non-limitingexample, ID for transformed logical data objects stored per CIFS storageprotocol may be generated in accordance with time of generation, CIFScommand parameters (e.g. tid, pid, uid, mid) and file name; for NFSstorage protocol the ID may be generated in accordance with time ofgeneration, NFS File-Handle, NFS FSID—FileSystemID and NFS FileID; forSCSI protocol the ID may be generated in accordance with DeviceIDcharacterizing a respective LUN.

The transformation system examines the received “write” requests inorder to recognize (31) addressed logical data objects and, accordingly,ID of respective transformed LOs. The transformation system furtherderives from received “write” requests data chunks obtained with therequest, respective offsets and lengths; and transforms 32 (e.g.compresses, encrypts, compresses & encrypts, etc.) the obtained datachunks into transformed data chunks, each transformed data chunkcorresponding to a respective received write request.

The teachings of the present invention are applicable to transformedlogical data objects accommodating transformed data chunks in a logform, journal form or any other form of sequential accommodation.

In addition to the transformed data, each transformed data chunkcomprises information in respect of an offset of the original data chunkwithin the logical data object and size of the original data chunk. Byway of non-limiting example, if transformed data chunks are accommodatedin a log form, such information may be held in a header of eachrespective log. The log header may also comprise transformation-relatedinformation (e.g. initialization vector, key ID, etc.).

Among the “write” requests received during certain period of time, theremay be a plurality of requests addressed to the same LO. These “write”requests may be initiated by different clients (workstations, servers,applications, etc.) and/or as multiple “write” requests by the sameclient or maintenance unit. The transformation system identifies therequests addressed to the same LO, and, as will be further detailed withreference to FIG. 4, groups (33) the requests and respective transformeddata chunks related to the same LO whilst keeping the transformed datachunks in the order corresponding to the order of the respectivereceived requests.

The grouping is provided until accomplishment of certain predefinedcriteria (e.g. until a certain period of time has elapsed, and/or untiltotal size of the transformed chunks in the group exceeds a predefinedsize, and/or until achieving a pre-defined number of grouped requests,etc.). The grouping may involve additional criteria as, for example,grouping may be provided only for multiple “write” requests obtainedfrom a single client or predefined users (or user groups), and/orpredefined applications, etc. “Write” requests initiated by clients andby the maintenance unit(s) may be grouped together or separately.

The transformation system further generates (34) grouped “write” requestto the storage system for writing the resulting group of the processeddata chunks to the storage. Upon writing to the storage, the groupeddata chunks are sequentially accommodated in the respective transformedlogical data object in accordance with the order these and previouschunks were received.

After the group of processed data chunks is written at the storagelocation, the transformation system receives an acknowledging responsefrom the storage system. The request mapping module 24-2 recognizesacknowledging responses with regard to the grouped “write” request amongresponses received from the storage system. The logical object manager24 further multiplies the response, and enables sending (35) respectiveacknowledgements to the sources of all requests in the group (e.g.workstation, workstation+application, maintenance unit, etc.) whoserequest has been fully accomplished.

The transformation system further provides mapping (36) of the storeddata, the mapping facilitating one-to-one relationship between dataderived from the received “write” requests (obtained data chunks) anddata to be read from the respective stored transformed logical object.

By way of non-limiting example, the transformed logical data object (LO)may be obtained as follows:

-   -   a) in response to a respective request, creating in the storage        device a transformed logical data object comprising a header and        one or more allocated accommodation sections with predefined        size;    -   b) processing one or more obtained variable size data chunks        corresponding to the transforming logical data object thus        giving rise to the transformed data chunks, wherein at least one        of said transformed data chunks comprises transformed data        resulting from said processing;    -   c) sequentially accommodating the transformed data chunks into        said accommodation sections in accordance with an order said        chunks received, and    -   d) facilitating mapping between the data in the transforming        logical data object and the data accommodated in the        accommodation sections.

The accommodation sections serve as atomic elements oftransformation/de-transformation operations during input/outputtransactions on the logical data object.

The processing may comprise one or more techniques of datatransformation (e.g. compression, encryption, compression together withencryption, etc.) These techniques may be coordinated in time, mayconcurrently execute autonomous sets of instructions, and be provided ina manner facilitating preserving the sequence of processing andaccommodating the processed data chunks.

The mapping may be provided with the help of at least one index sectionconstituting a part of the transformed logical data object, the indexsection comprising at least one record holding at least informationrelated to transformed data chunks accommodated in at least oneaccommodation section and indication of physical storage locationpertaining to the accommodation section. The information related to thetransformed data chunk may comprise, for example, offset and size in thenon-transformed LO, association with the accommodation section, etc.)

Certain embodiments of the above exemplified method of generating atransformed logical data object for storage in a storage device andrespective writing and reading operations are detailed in InternationalApplications WO2007138600, WO2007138601 and WO2007138602 published Dec.12, 2007 and assigned to the assignee of the present application.

In a case of transformed LO accommodating transformed data chunks in oneor more accommodation sections (AS) as described above, thetransformation system, upon generating (34) grouped “write” request,writes the group of transformed chunks to an active accommodationsection (AS accommodating the last written processed data chunk). If theactive accommodation section does not comprise enough free space toaccommodate the grouped data chunks (e.g. writing operation fails ontarget buffer overflow, group size more than said free space, etc.), thetransformation system requests the session manager to allocate a newaccommodation section.

Optionally, if the free space in the active AS is insufficient toaccommodate the entire group of chunks, but meets a predefined criteria(e.g. more than predefined size, more than predefined ratio of entirerequired space, etc.), the transformation system splits the group intotwo parts, writing one in the active AS and the second in the new AS.

If a group has been split between accommodation sections, thecorresponding response is sent to the originating addresses only afterthe transformation system receives from the storage systemacknowledgement of storing the entire group.

Mapping (36) may be provided by updating the index section. Such updatemay be provided substantially in parallel with acknowledgement, whenallocating the new AS and/or closing the LO and/or in accordance withother predefined rules. The index section update includes adding recordsrelated to the new transformed data chunks, and corresponding updatingprevious records related to live and/or outdated data comprised in thecorresponding range.

FIG. 4 illustrates a generalized flowchart of handling a write requestin a “write” request queue in accordance with certain embodiments of thepresent invention. Upon identifying (31) a logical data object addressedin a “write” request received from a client or a maintenance unit, thelogical object manager 24 appends (41) the request to a pending “write”requests queue associated with the identified LO. The pending writesunit 24-1 is configured to open and handle the queues of pending “write”requests addressed to the same LOs. The obtained data chunk is processed(42) by the transformation unit 26 into transformed data chunk andaccommodated in an output buffer associated with the identified LO. Thetransformation unit further updates the request mapping module 24-2 withregard to association between the transformed data chunk and respectiveLO. Operations 41 and 42 may be provided sequentially of in parallel.

The logical object manager further checks (43) if there are requests inthe pending “write” requests queue associated with the identified LO,and if “No”, facilitates (44) sending the transformed data chunk forwriting in the storage device. If the queue comprises one or morerequests addressed to the same identified LO, the buffer manager checks(45) the status of the output buffer associated with said LO. If thetotal size of transformed data chunks related to the identified LO andaccommodated in the output buffer matches certain criteria (i.e.respective buffer is full), the accommodated transformed data chunks arecombined in the grouped “write” request. The generated grouped “write”request is further sent (48) to the storage device. Transformed datachunks in the grouped “write” request are combined in the ordercorresponding to the order of the respective received requests in thequeue, and, upon writing, sequentially accommodated in the transformedLO in accordance with the same order. If the “write” request (30) is notfully processed because of the full buffer, it may be appended to a newqueue or, optionally, may be truncated and split (47). If the buffer isnot full, the transformed data chunk is kept (46) in respective outputbuffer and is sent to the storage device upon accomplishment of acertain criteria (e.g. elapsed runtime, filling up the buffer withtransformed data chunks corresponding to other “write” requests to thesame identified LO, etc.) as a part of the respective group oftransformed data chunks.

As long as there are requests in the pending writes queue associatedwith a given LO, they are processed by the transformation unit into theoutput buffer and grouped as described above.

The grouping illustrated in FIG. 4 is provided until achieving apredefined size of a respective buffer. As was detailed with referenceto FIG. 3, the grouping may be provided until accomplishment of anothercriteria and/or combination thereof.

It is to be understood that the invention is not limited in itsapplication to the details set forth in the description contained hereinor illustrated in the drawings. The invention is capable of otherembodiments and of being practiced and carried out in various ways.Hence, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting. As such, those skilled in the art will appreciatethat the conception upon which this disclosure is based may readily beutilized as a basis for designing other structures, methods, and systemsfor carrying out the several purposes of the present invention.

It will also be understood that the system according to the inventionmay be a suitably programmed computer. Likewise, the inventioncontemplates a computer program being readable by a computer forexecuting the method of the invention. The invention furthercontemplates a machine-readable memory tangibly embodying a program ofinstructions executable by the machine for executing the method of theinvention.

Those skilled in the art will readily appreciate that variousmodifications and changes can be applied to the embodiments of theinvention as hereinbefore described without departing from its scope,defined in and by the appended claims.

The invention claimed is:
 1. A method of operating a transformationsystem operatively coupled to a plurality of servers and to a pluralityof storage devices via a storage network and operable to write aplurality of data chunks to a plurality of transformed logical dataobjects (LOs) stored in the storage device, each given transformedlogical data object corresponds to a respective original logical dataobject and configured to sequentially accommodate a plurality ofvariable size data chunks corresponding to the original LO, transformedby the transformation system and accommodated in accordance with anorder in which said chunks are received; the method comprising:transmitting, by the plurality of servers, a plurality of data chunksand a plurality of “write” requests to write the plurality of datachunks to the plurality of storage devices, each data chunk associatedwith a respective “write” request; identifying among the plurality of“write” requests each “write” request addressed to the same transformedLO; deriving data chunks corresponding to each identified “write”request and processing the derived data chunks, thereby giving rise totransformed data chunks; grouping the transformed data chunks that areaddressed to the same transformed LO in a respective group and inaccordance with the order the “write” requests were transmitted, thusgiving rise to a plurality of groups of transformed data chunks;determining if a respective transformed LO for accommodating aparticular group of transformed data chunks in a respective storagedevice includes sufficient free space to accommodate the particulargroup of transformed data chunks; if the respective transformed LOincludes sufficient free space to accommodate the particular group oftransformed data chunks: generating a grouped “write” request to therespective transformed LO whilst keeping the order the data chunks, andwriting the particular group of transformed data chunks to therespective transformed LO; if the respective transformed LO includesinsufficient free space to accommodate the particular group oftransformed data chunks: determining if the particular group oftransformed data chunks meets a predefined criteria related to a size ofthe particular group of transformed data chunks, if the size of theparticular group of transformed data chunks meets the predefinedcriteria: allocating a new transformed LO in the respective storagedevice, splitting the grouped “write” request into a plurality of parts,generating a first grouped “write” request to the respective transformedLO, the first grouped “write” request comprising a first part of theplurality of parts, generating a second grouped “write” request to thenew transformed LO, the second grouped “write” request comprising asecond part of the plurality of parts, writing the first part of theplurality of parts to the respective transformed LO, and writing thesecond part of the plurality of parts to the new transformed LO, and ifthe size of the particular group of transformed data chunks fails tomeet the predefined criteria: allocating the new transformed LO, andwriting the particular group of transformed data chunks to the newtransformed LO; and mapping between one of the data in the original LOand the data accommodated in the respective transformed LO, the data theoriginal LO and the data accommodated in the respective transformed LOand in the new transformed LO and the data the original LO and the dataaccommodated in the new transformed LO in a manner facilitatingone-to-one relationship between the data in the transformed data chunksand the data to be read from the corresponding one of the respectivetransformed LO, the respective transformed LO and new transformed LO,and the new transformed LO.
 2. The method of claim 1 further comprising:(a) obtaining from the storage device an acknowledging response withregard to each grouped “write” request; (b) multiplying the obtainedacknowledging response, and (c) sending respective acknowledgements toeach source which initiated at least one identified “write” requestcorresponding to each respective group of transformed data chunks. 3.The method of claim 1 wherein at least part of the identified “write”requests is obtained by intercepting by the transformation system aplurality of “write” requests addressed by one or more servers amongsaid plurality of servers to at least one storage device.
 4. The methodof claim 1 wherein at least part of the identified “write” requests isgenerated by the transformation system during a maintenance and/oroptimization process.
 5. The method of claim 3 wherein, responsive tointercepting a request addressed to the respective transformed LO, thetransformation system enables identifying the respective transformed LOin a manner transparent to a server.
 6. The method of claim 3 whereinidentifying two or more “write” requests addressed to the sametransformed LO comprises recognizing, by the transformation system,unique identification of each respective transformed LO.
 7. The methodof claim 1 wherein the transformed logical data object accommodatestransformed data chunks in a log form.
 8. The method of claim 1 whereinthe transformed logical data object accommodates transformed data chunksin a journal form.
 9. The method of claim 1 wherein transformed datachunks comprise information with respect to an offset of a respectivederived data chunk within the original logical data object andinformation with respect to a size of said respective derived datachunk.
 10. The method of claim 1 wherein the mapping is provided withthe help of at least one index section constituting a part of thetransformed logical data object, the index section comprising at leastone record holding at least information related to transformed datachunks.
 11. The method of claim 1 wherein the transformed data chunksare accommodated in the in one or more accommodation sections serving asatomic elements of transformation/de-transformation operations duringinput/output transactions on each transformed logical data object.
 12. Amethod of writing data to a plurality of transformed logical dataobjects (LOs) stored in a plurality of storage devices operable with atleast one storage protocol, each transformed LO comprising a header, anindex section and one or more accommodation sections with a predefinedsize, said sections sequentially accommodating transformed data chunksresulting from transformation of a respective logical data object, themethod comprising: receiving, from a plurality of servers, a pluralityof data chunks and a plurality of “write” requests to “write” theplurality of data chunks to the plurality of storage devices, each datachunk associated with a respective “write” request; identifying among aplurality of “write” requests each “write” request addressed to the sametransformed LO; deriving data chunks corresponding to each identified“write” request and processing the derived data chunks, thereby givingrise to transformed data chunks; grouping the transformed data chunksthat were transmitted by the same server in a respective group and inaccordance with the order the “write” requests were received, thusgiving rise to a plurality of groups of transformed data chunks;determining if a respective transformed LO for accommodating aparticular group of transformed data chunks in a respective storagedevice includes sufficient free space to accommodate the particulargroup of transformed data chunks; if the respective transformed LOincludes sufficient free space to accommodate the particular group oftransformed data chunks: generating a grouped “write” request to therespective transformed LO whilst keeping the order the data chunks, andwriting the particular group of transformed data chunks to therespective transformed LO; if the respective storage device includesinsufficient free space to accommodate the particular group oftransformed data chunks: determining if the particular group oftransformed data chunks meets a predefined criteria related to a size ofthe particular group of transformed data chunks, if the size of theparticular group of transformed data chunks meets the predefinedcriteria: allocating a new transformed LO in the respective storagedevice, splitting the grouped “write” request into a plurality of parts,generating a first grouped “write” request to the respective transformedLO, the first grouped “write” request comprising a first part of theplurality of parts, generating a second grouped “write” request to thenew transformed LO, the second grouped “write” request comprising asecond part of the plurality of parts, writing the first part of theplurality of parts to the respective transformed LO, and writing thesecond part of the plurality of parts to the new transformed LO, and ifthe size of the particular group of transformed data chunks fails tomeet the predefined criteria: allocating the new transformed LO, andwriting the particular group of transformed data chunks to the newtransformed LO; and updating the index section in a manner facilitatingone-to-one relationship between the data in transformed data chunks andthe data to be read from the corresponding one of the respectivetransformed LO, the respective transformed LO and new transformed LO,and the new transformed LO.
 13. The method of claim 12 furthercomprising: (a) obtaining an acknowledging response from each storagedevice; (b) multiplying the obtained acknowledging response, and (c)sending respective acknowledgements to each source which initiated atleast one identified “write” request corresponding to each respectivegroup of transformed data chunks.
 14. The method of claim 12 wherein atleast part of the identified “write” requests is obtained byintercepting by the transformation system a plurality of “write”requests addresses by one or more servers among said plurality ofservers to at least one storage device.
 15. The method of claim 12wherein at least part of the identified “write” requests is generated bythe transformation system during a maintenance and/or optimizationprocess.
 16. The method of claim 14 wherein, responsive to interceptinga “write” request addressed to the respective transformed LO, thetransformation system enables identifying the respective transformed LOin a manner transparent to a server.
 17. The method of claim 16 whereinidentifying two or more “write” requests addressed to the sametransformed LO comprises recognizing, by the transformation system,unique identification of each respective transformed LO.
 18. Atransformation system capable of transforming a logical data object forstorage in a storage device, said system operatively coupled to aplurality of servers and to a plurality of storage devices configured toaccommodate data transformed by the transformation system, the systemcomprising: means for creating in said at least one storage device aplurality of transformed logical data objects (LOs), each giventransformed LO corresponding to a respective original logical dataobject (LO) and configured to sequentially accommodate a plurality ofvariable size data chunks corresponding to the original LO, transformedby the transformation system and accommodated in accordance with anorder in which said data chunks are received; means for transmitting, bythe plurality of servers, a plurality of data chunks and a plurality of“write” requests to write the plurality of data chunks to the pluralityof storage devices, each data chunk associated with a respective “write”request; means for identifying among a plurality of “write” requestseach “write” request addressed to the same transformed LO; means forderiving data chunks corresponding to each identified “write” requestand processing the derived data chunks, thereby giving rise totransformed data chunks; means for grouping the transformed data chunksthat were transmitted by the same server in a respective group and inaccordance with the order the “write” requests were received, thusgiving rise to a group of transformed data chunks; means for determiningif a respective transformed LO for accommodating a particular group oftransformed data chunks in a respective storage device includessufficient free space to accommodate the particular group of transformeddata chunks; if the respective transformed LO includes sufficient freespace to accommodate the particular group of transformed data chunks:means for generating a grouped “write” request to the respectivetransformed LO whilst keeping the order the data chunks, and means forwriting the particular group of transformed data chunks to therespective transformed LO; if the respective transformed LO includesinsufficient free space to accommodate the particular group oftransformed data chunks: means for determining if the particular groupof transformed data chunks meets a predefined criteria related to a sizeof the particular group of transformed data chunks, if the size of theparticular group of transformed data chunks meets the predefinedcriteria: means for allocating a new transformed LO in the respectivestorage device, means for splitting the grouped “write” request into aplurality of parts, means for generating a first grouped “write” requestto the respective transformed LO, the first grouped “write” requestcomprising a first part of the plurality of parts, and means forgenerating a second grouped “write” request to the new transformed LO,the second grouped “write” request comprising a second part of theplurality of parts, means for writing the first part of the plurality ofparts to the respective transformed LO, and means for writing the secondpart of the plurality of parts to the new transformed LO, and if thesize of the particular group of transformed data chunks fails to meetthe predefined criteria: mean for allocating the new transformed LO, andmeans for writing the particular group of transformed data chunks to thenew transformed LO; and means for mapping between the data in theoriginal LO and the data accommodated in the respective transformed LOin a manner facilitating one-to-one relationship between the data in thetransformed data chunks and the data to be read from the cones ondin oneof the respective transformed LO, the respective transformed LO and newtransformed LO, and the new transformed LO.
 19. The system of claim 18further comprising: (a) means for obtaining an acknowledging responsefrom each storage device; and (b) means for multiplying the obtainedacknowledging response, and sending respective acknowledgements to eachsource that initiated at least one identified “write” requestcorresponding to each group of transformed data chunks.
 20. The systemof claim 18 further comprising means for intercepting at least part ofthe plurality of “write” requests addresses-addressed by one or moreservers among said plurality of servers to said plurality of storagedevices.
 21. The system of claim 18 further comprising a maintenanceunit operable to generate, during a maintenance and/or optimizationprocess, one or more “write” requests addressed to one of the pluralityof storage devices.
 22. A computer memory comprising computer programcode means for performing all the steps of claim 1 when said computerprogram code is executed by a processor.
 23. A computer memory asclaimed in claim 22 embodied on a non-transitory computer readablemedium.